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The Florida Online Reading Professional Development (FOR-PD) program is funded by the 
Florida Department of Education (DOE) and is housed at the University of Central Florida (UCF). FOR- 
PD is an online staff development project designed to help teachers improve reading instruction for learners 
in grades preK-12. Developed collaboratively with literacy and technology experts, school districts, 
professional organizations, and teacher educators across the state of Florida, the project functions as a 
primary statewide delivery mechanism for improving teaching methods in reading instruction to preK-12 
teachers. 

As indicated in the Request for Proposal (RFP) from the Florida Department of Education (FL 
DOE), for example, “feedback and leader-peer response” and “monitoring of assignments” were considered 
essential. In response to the RFP, the UCF proposal for FOR-PD highlighted the role of facilitators, 
asserting in the grant proposal text, “Feedback by facilitators is critical to the performance of participants.” 
In addition, the discussion boards were also depicted as critical factors for successful delivery of online 
learning by both documents. 

Prior research has emphasized facilitator’s roles in online education or training courses, 
“facilitating online dialogue, community, and ultimately, education” (Collison, Elbaum, Haavind, & 

Tinker, 2000). Meanwhile, Lieblein (2000) explored the role of threaded discussion board as s critical 
factor for successful delivery of online programs based on nearly 10 years of academic and administrative 
experience with online programs. On the other hand, qualitative analysis could complement and be used in 
conjunction with the quantitative methods by explaining reasons for observed differences (Gunawardena, 
Lowe, & Carabajal, 2000). 



Recruiting Facilitators 

We relied on three core criteria for selecting facilitators for FOR-PD. First, the facilitator needed 
to have strong content knowledge in reading. Second, we sought online facilitators who had experience as 
literacy leaders and literacy experts. Third, we looked for facilitators who expressed desire to learn along 
with us about helping preK-12 teachers develop their reading knowledge and expertise. Given the novelty 
of this large-scale high profile state online project, prior experience with the Internet was not mandatory. 
However, we knew that some facilitators were reasonably comfortable in online learning since they had 
been involved in it before as students or facilitators. 

FOR-PD Facilitators 

FOR-PD facilitators play a vital role in developing and maintaining an online professional 
development program that is effective, efficient, and supports the realization of the FOR-PD project 
objectives. The primary purpose of a FOR-PD facilitator is to interact with FOR-PD course participants. 
This translates to encouraging and replying to email messages and discussion postings, providing feedback 
on assignments, and being the "point person" for answering their questions. They must also be responsive 
to individual district requests and needs. 

A facilitator in the online environment must possess a unique set of skills to perform effectively. Some of 
the basic criteria for a person to be successful as an online facilitator include the following. Facilitators 
must be able to create a supportive environment where all students feel comfortable participating and 
especially where students know that their facilitator is accessible. Facilitators should give students timely 
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quality feedback on student contributions to discussions, assignments, and quizzes. Facilitators should keep 
students advised of their progress respect to the course evaluation process on a regular basis. Facilitators 
should feel comfortable communicating in writing. The face-to-face contact traditionally available in a 
classroom setting is not available in the online learning process. The ability to verbally communicate is 
replaced with a keyboard. Facilitators must be comfortable communicating in writing because that is the 
fundamental process of online learning. The facilitator is the primary person participants interact with who 
provides the human factor. Facilitators should be experienced and well trained in online learning. This 
includes: sending and receiving email; using discussion boards; using chat tools; using a web browser- 
Netscape or Internet Explorer. 

Facilitator Training and Certification Course 

To become a certified FOR-PD facilitator, interested educators must complete an online 
application and possess the following qualifications: 1) successful completion (80% mastery or above) of 
the FOR-PD course; 2) a minimum of three years teaching experience; 3) master’s degree in reading or 
other related areas; 4) advanced knowledge of research-based reading strategies; 5) ability to provide 
explicit instruction in the following elements of reading as they apply to appropriate grades: phonemic 
awareness, phonics, fluency, vocabulary, and comprehension; 6) ability to systematically use effective 
reading strategies that have been tested and have a record of success; and 7) identified by school or district 
as a reading/literacy leader. In addition to these requirements, successful completion of the FOR-PD 
Facilitator Training and Certification Course is also required. The FOR-PD Facilitator Training and 
Certification Course is a 25-hour online professional development course intended to ensure that our 
facilitators have the knowledge and skills they need to become successful online class facilitators for the 
Florida Online Reading Professional Development Course. The course consists of the five lessons that 
encompass an introduction to FOR-PD, details about the project and goals of FOR-PD, information on 
online learning, and support options to facilitators 

Following completion of the FOR-PD Facilitator Training and Certification Course, facilitators 
are expected to demonstrate mastery of the following skills: 1) describe the FOR-PD course and the goals 
of the course; 2) identify advantages of online learning; 3) identify potential disadvantages of online 
learning and describe at least one way each disadvantage can be addressed; 4) identify the role of the online 
class facilitator; 5) describe techniques for facilitating an online course; and 6) identify and use online tools 
such as chat, discussion boards, email, and grade books. 

There is no charge to take the FOR-PD Facilitator Training and Certification Course. An 
electronic certificate (pdf) is emailed to participants upon successful completion, their district staff 
development office is notified, and they are then added to the pool of certified FOR-PD facilitators. 
Completion of the FOR-PD Facilitator Training and Certification course and certification as a FOR-PD 
facilitator does not guarantee employment as a facilitator. For the most part, facilitators are selected to 
facilitate by school districts from the pool of qualified facilitators. Many school districts have a “favorite” 
facilitator or two that they assign to facilitate again and again. These are generally reading specialists, 
reading coaches, or literacy leaders in the district with particular knowledge of the unique qualities of the 
district, its reading programs, teachers, and student population. 

Facilitator Support 

Facilitators receive assistance from the FOR-PD office to support them in their roles as online mentors 
for participants. Support in place includes a facilitator manual, performance support tools, electronic 
newsletters, facilitator forums, monthly chats, and supervision. 

Facilitator Manual 

One of the most important documents available to facilitators, the Facilitator Manual, outlines 
specific tasks facilitators must complete before their section of the FOR-PD course begins. Such tasks 
include getting access to the course, getting participants’ names, posting the first discussion, and sending a 
welcome message to participants. The facilitator manual also offers a detailed description of facilitator tasks 
to be completed throughout the course including reminders to monitor participants’ progress, grading 
assignments, offering feedback, handling course “no shows,” and those who fall behind. The Facilitator 
Manual offers detailed explanations for after-course tasks such as sending a wrap-up message, thanking 
participants for their participation, and instructions for notifying school district and FOR-PD offices that the 
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course has ended. The manual also describes the various support mechanisms available to facilitators and 
outlines the responsibilities of key stakeholders including participants, school districts, and the FOR-PD 
office. Conditions of employment are carefully explained and tutorials for WebCT chat, discussion boards, 
email tools, and grade book tools are provided. The manual’s appendix offers sample messages for a variety 
of purposes including welcoming participants to the course, helping those with login trouble, notice to those 
lagging behind, and a sample message to thank participants for their participation. 

Performance Support Tools 

FOR-PD course facilitators have access to a variety of performance support tools to help them in 
their role as course facilitator. Specifically requested by course facilitators, printable rubrics streamline 
assignment grading for busy facilitators who prefer to use a paper-based method of grading course 
assignments. The Excel grade book template offers a convenient method of grading for facilitators more 
comfortable with storing grades electronically. Developed at the request of course facilitators, model 
postings provide examples for facilitators to share with participants as needed and in a way controlled by 
the facilitators. The model postings also help facilitators calibrate their own grading standards. Some 
facilitators have requested a course completion certificate for participants in their sections. This optional 
component is available to facilitators as needed through the Facilitator Forum. The end-of-course checklist 
serves as a convenient reminder to facilitators about the specific tasks that must be carried out to assure 
proper in-service credit for participants and prompt payment to course facilitators. 

Electronic Newsletter for Facilitators 

Entitled Facilitation With Felicity (FFF*), the facilitator newsletter helps keep course facilitators 
informed of the latest literacy news and events from around the state and from the FOR-PD project. The 
FFF* highlights one literacy strategy each month and offers a variety of tips and tricks to keep facilitators’ 
skills polished. One section focuses on “fine facilitation’’ to highlight and promote desirable facilitator 
actions. The “Dear Felicity’’ column answers facilitators’ questions about handling problems, dealing with 
challenging participants, and keeping participants involved in the course. The FFF* also announces awards, 
contests, and conferences of interest to facilitators and highlights a wide variety of literacy and professional 
resources likely to be useful to FOR-PD’s facilitators. 

Facilitator Forum 

Housed on the FOR-PD course server, the Facilitator Forum is a series of discussion boards 
offering 24/7 access for facilitators to interact with each other to share information and ideas about the 
FOR-PD course, to ask for help from others, and to share successes. Links to FOR-PD course content, the 
FFF*, the Facilitator Manual, and optional course completion certificate offer convenient access for 
facilitators. Specific discussion areas include a place to meet fellow facilitators, to ask for and offer help, 
hints, and advice, to make suggestions for an upcoming FFF*, and to share success stories. The Facilitator 
Coffeehouse discussion board enables facilitators to interact with each other on matters unrelated to the 
FOR-PD course, but likely to be of general interest. Finally, there is a discussion area specifically for 
facilitators to discuss issues related to each of the 14 FOR-PD lessons. 

Monthly Chats 

Monthly facilitator chats serve to keep facilitators in touch with each other and the FOR-PD office. 
Chats feature discussions with subject matter experts and cover a range of literacy-related topics. 

Facilitators often discuss the mechanics of facilitating a course, recent changes to the course or participate 
in an open forum facilitated by FOR-PD staff to answer questions. 

Personnel 

All FOR-PD staff members assist facilitators in answering questions or solving problems. The 
FOR-PD Help Desk is available for technical support to facilitators and participants alike. One full-time 
staff member is dedicated to assisting facilitators perform their duties. The Facilitator Support Specialist 
monitors discussions on the Facilitator Forum, providing answers as necessary and identifying issues 
raised in the Facilitator Forum that need to be addressed more broadly by other FOR-PD staff members 
and sharing suggestions to improve the project. Additional responsibilities include answering email from 
facilitators, writing the FFF* and coordinating monthly chat sessions, as well as answering Help Desk 
calls. 
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The Facilitator Support Specialist responds to emails received from the forpdfac email account 
for day-to-day, routine implementation questions, and is also available to support facilitators through the 
duration of their facilitation experience, including sending reminders and instructions to facilitators as 
course sections begin and end. After the course has finished, Facilitator Support Specialist sends follow-up 
information and reminders and processes payroll paperwork. 

Facilitator Supervision 

In addition to the required training and certification, facilitators are closely monitored throughout 
their facilitation of the FOR- PD course. Each section of the course is monitored at three points: 1) the 
beginning of the course (Lesson 1); 2) the midpoint (Lesson 5-8); and 3) at the end (Lesson 14). The 
monitoring helps to ensure that facilitators are creating a supportive environment where all students feel 
comfortable participating and especially where students know that their facilitator is accessible; giving 
students timely quality feedback on student contributions to discussions, assignments, and quizzes; and 
keeping students advised of their progress in the course on a regular basis. 

The FOR-PD project has been exceptionally well received by school districts, universities, 
administrators, and teachers throughout the state. We believe that our facilitators and the training, 
certification, and support that we provide to them has been critical to the success of the FOR-PD project. 
Currently, FOR-PD has 200 certified facilitators located throughout the state. Of these 200, 111 reported 
having taken an online course previously, however only 23 had ever taught or facilitated an online course 
before coming to FOR-PD. 390 sections of the FOR-PD course have been offered since January 21, 2003 
and 139 facilitators have facilitated a section within the last year. 

FOR-PD Evaluation 

The outside interim report of the first year of the FOR-PD project and course, drawn from various 
sources including narrative reports from facilitators, surveys of participants at the end of the course, and 
follow-up telephone interviews with administrators, reported the following. Over 87% of FOR-PD 
participants indicated they would make changes and/or additions to classroom reading instruction as a 
result of FOR-PD. Over 90% (93%) of participants indicated that the value of reading strategies introduced 
in FOR-PD was excellent or good. Approximately 97% of participants indicated FOR-PD was excellent or 
good in covering the state and national reading initiatives, with nearly three-fourths of participants (73%) 
indicating FOR-PD covered the reading initiatives to an excellent degree. Over 90% of participants 
indicated that FOR-PD has contributed to their knowledge of effective reading theory, research, and 
instructional practice to an excellent or good extent. The extent FOR-PD contributed to understanding 
student needs and instructional adaptations for struggling readers to an excellent or good extent was 89% 
with over one-half indicating excellent (52%). Over 90% of participants rated the support from their 
facilitator as excellent (74%) or good (17%). 

For the purpose of the qualitative evaluation of FOR-PD in phase II of the project, internal and 
external documents were reviewed. 1 Hundreds of pages of qualitative data were collected and analyzed 
from the discussion boards of facilitators, with a focus on what contributions FOR-PD facilitators have 
made to the program, how they have experienced the program, and particularly, how they like the changes 
and revision of the FOR-PD course since summer 2003. 

FOR-PD Pilot Summary Report 

The FOR-PD Course was piloted September 16-27, 2002. The pilot summary report focused on 
feedback from participants and concluded that “Overall, the feedback from the participant was extremely 
positive . . .Teachers indicated that the course provided aspects that they would utilize in their classroom 
instruction”. This interim report drew from various sources including narrative reports from facilitators, 
surveys of participants at the end of the course, and follow-up telephone interviews with administrators. 

FOR-PD Courses, Participants and Facilitator’s Discussion Board 

The first sections of the FOR-PD course were launched on January 21, 2003. A total of 4,570 
participants had enrolled and 2,045 had completed FOR-PD prior to beginning phase 2 (September 2003). 
Of this number, 4,504 participated as employees of Florida school districts and 66 participated through a 
Florida college or university. The latter participants may include students and college/university faculty. 
During fall 2003, approximately 2,200 additional students enrolled in FOR-PD. 

In February 2003, a Facilitator Forum was established using the WebCT discussion board. The 
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primary purpose for this discussion board was to provide a support mechanism for facilitators to ask for, 
and provide advice to other facilitators. A total of 287 individuals were accepted to facilitate in FOR- PD, 

70 (25% of those accepted) of which were facilitating at least one section of FOR-PD as of September 
2003. The average years of teaching experience for facilitators was nearly 18 years with one-half boasting 
at least 17 years of experience. Approximately 56% of facilitators had taken an online course, however 
only 12% had previous experience in facilitating online. 

Facilitator Focus Group Report 

A facilitator focus group was conducted in May 2003 at UCF. Twenty-seven facilitators attended 
the focus group meeting. The focus group report provided a broad picture of the FOR-PD system from 
facilitator’s perspective. The general tone of responses from the focus group was very positive. Facilitators 
appreciated the current research-based content, the database of materials and resources developed as part of 
FOR-PD. However, issues and concerns were raised in the focus group, too. For example, some 
participants had copied from each other and had submitted work that had been directly copied from the 
Internet. In addition, facilitators listed the WebCT grade book as their No. 1 technology problem. Finally, 
the report gave eight recommendations for full-time FOR-PD staff to consider and stated that many of 
those (concerns and issues) had been addressed through the work of the editing team several weeks after 
the focus group meeting and the current course revision underway (before the beginning of phase 2)” 

Methods 

The site and major data source for the current qualitative evaluation was the facilitators’ 
discussion board postings from fall September 2003. There were 161 postings as of December 31, 2003, 
including topics such as: meeting peer facilitators and introducing their backgrounds, making comments, 
extending greetings to each other for the new semester 2003 fall, offering help, hints and advice, sharing 
success stories of FOR-PD participant learning, providing suggestions the FOR-PD facilitator’s electronic 
newsletters, providing general ideas and suggestions, posting messages related or unrelated to FOR-PD 
course, and discussing Lesson 1 to Lesson 14. The data used for the evaluation consisted of 120 messages 
posted by the facilitators out of the total of 161 on the discussion board since the beginning of phase 2. As 
the major source of data for the present evaluation, facilitator’s discussion board met some requirements of 
the RFP and a few focuses indicated in the UCF proposal for FOR-PD. 

Robert Yin's book. Case study research: design and methods (2nd ed.) was used for the design for this 
qualitative evaluation and one of its dominant mode of case study analysis, the combination of ‘pattern - 
matching’ and ‘time -series analysis’ was applied to analyze and explain facilitators’ perceptions and 
experiences of FOR-PD. Moreover, with the qualitative software Nvivo Revision [1.3], automatic coding of 
the data was used in addition to hand coding and various codes were developed (see details in Appendix: 
Data Analysis). 



Findings 

From the Facilitators’ discussion board postings, two major categories emerged, messages 
conveying facilitator’s comments and messages conveying facilitator’s activities. Meanwhile, the 
frequency of postings on the discussion board varied largely from month to month at different data points: 
September, October, November and December. As Table 1 shows, out of the total 161 posts including 
coordinator or instructor’s messages, there are 95 more in the first half of the semester (September and 
October) than the later half (November and December). 

Table 1 Monthly Posts in Facilitator’s Discussion Board Phase 2 



Month 


Number of Posts 


September 


89 


October 


39 


November 


20 


December 


13 
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Facilitators’ Comments 

Operationally, ‘encouragement’ included expressions like ‘look forward to new session’, ‘like the 
changes since summer’, ‘learn a lot’ from FOR-PD, and etc, while ‘criticism’ connoted ‘frustrating about 
participants who dropped out’, ‘unavailable assistance’ and etc. In terms of comments (see Table 2), the 
first half of the semester had a contrastingly larger amount of ‘encouragement’ from facilitators than the 
later half of the semester, with a ratio of 57 to three. Likewise, 15 negative messages appeared in the first 
half while none in the later half. In general, there were a lot more ‘encouragement’ than ‘criticism’ during 
the whole section. 

‘Encouragement’ messages were divided into six categories (Table 3). Specifically, a lot of 
facilitators expressed cheerfulness, looked forward to new session and liked the changes of layout and 
content since summer. 



Table 2 Monthly Posts of Comments in Facilitators’ Discussion Board Phase 2 



Comments 




encouragement 


criticism 


September 


50 


8 


October 


7 


7 


November 


2 


0 


December 


1 


0 



Table 3 Facilitators’ Comments in the Discussion Board in Phase II 



Facilitators’ Comments 


Encouragement 


Sept. 


Oct. 


Nov. 


Dec. 


look forward to new session and like the changes, including 
the scoring rubric 


X 


X 




X 


enjoy being a facilitator, and show pride in completion rate 


X 


X 






like FOR-PD course 


X 








appreciate facilitator discussion boards 


X 


X 






Identify with participants' encouragement 


X 








share influence of for-pd course in both schools and families, 
something beyond participants’ learning 


X 


X 


X 




Total Monthly No. 


50 


7 


2 


1 


Criticism 


Sept. 


Oct. 


Nov. 


Dec. 


frustration over participants’ low completion rate 


X 








complaints/confusion about technical problems, some caused 
by the changes in the new session 


X 


X 






empathy with participants’ frustration over assistance from 
FOR-PD project staff, such as help desk 


X 


X 






Total Monthly No. 


8 


7 


0 


0 



Following are quotes identified and extracted from the discussion board for ‘encouragement’ messages 
from facilitators. 

Iam excited to see the final version of the changes. I know the previews were impressive. Iam 
looking forward to the start of the new sections... 1 look forward to my next chance to facilitate. 
Until that time I plan on regular visits to the Discussion area to see what exciting things continue 
to transpire in the wonderful land of FOR-PD. ” 

“I look forward to the new format. So far, it looks great and seems so much more user friendly. Of 
course, I didn 't realize that the old course could be improved upon that much! ” 

“The summer course design was SO much easier to manage. I just wish that more teachers would 
take advantage of this wonderful opportunity. ” 

“The changes that were made for the fall are awesome! Thanks for all of the hard work FOR-PD 
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staff... you guys are great. ” 

Other facilitators enjoyed being a facilitator or liked the FOR-PD course itself. Following are 
examples. 

“Catherine. This is not a difficult course to facilitate. I did it for the first time this summer and I 
loved it. ” 

“I loved facilitating this summer. This is a terrific course. I've learned a lot”. 

“I took the FOR-PD course as a student last fall and was impressed... I found that the FOR-PD 
course offered that and a window to the ever-changing legislative directives. The more I learn the 
more 1 can share with my students, teachers and parents. ” 

“We had a third grade unit open up last week and I will be moving to the 3 rd grade unit on the 
18 th . I am very excited and plan on using much of what I have learned here in my class”. 

“It was a great experience for me to travel to different places around the state. It's shame that we 
really don't get to interact with more educators in different districts. That's one of the great 
benefits of this course. ” 

“The FOR-PD class has been a big part of my life since March. I carried my laptop cdl summer 
long as I traveled through the state with my daughter's softball team. I was not given a new 
section to facilitate and I will miss the interaction, the lesson discussions, and the daily 
information bank on literacy!” 

A few more appreciated the role of the facilitator discussion board. 

“Well I'm thankful for the facilitator discussion boards because I think this will be my lifeline. 

This is my first time facilitating a course & honestly I'm a little nervous. ” 

“This site will save you when you need really need it!” 

“I am looking forward to learn ingfrom all the experts who have already facilitated this course 
and I know I will get a lot of use out of this discussion board. ” 

A few facilitators identified with participants’ satisfaction with the course. 

“I look forward to the new look of the course and hope to hear many more wonderful things about 
this course. I had a participant show how much she enjoyed this course by making it part of her plan 
(lesson 14). She is going to be working with her administration to try to convince everyone that they 
need to take this course. That was a big “Wow” to me”. 

“I truly enjoy the notes I've received from course participants sharing how they've used strategies 
from the FOR-PD course in their classrooms. It's exciting to be a part of the process of having 
EVERY teacher become a reading teacher 1” 

“We have had a great response from our teachers... ” 

Moreover, some extremely positive themes that occurred in the discussion board, including the 
influence of FOR-PD course in schools and families, the impact beyond participants’ learning success. 

“Our SAC wrote the course into our School Improvement Plan this year, so all of our instructors 
who teach high school students will be taking the course. Several of our instructors who teach 
only adult students opted to take it also! ” 

“What a great idea! I am SAC coordinator at my school, along with all my other jobs. ..I think 
that writing the course into the SIP is a great idea. I'll have to look into it. ” 

“And YES, we are currently applying FOR-PD to Competency #2 in our district Reading 
Endorsement Plan. 1 am very happy to be involved in this professional development activity. I 
believe that we will have a better completion rate this fall than in the past. ” 

“ This was shared by one of my course participants.... ‘I also want to tell you that when I was doing 
the ABC brainstorming page for lesson 4 (I think), my 5 year old daughter was watching and 
asking what I was doing. She asked me to print a page for her, which I did, and she did all by 
herself. I am enclosing it in my literacy log. I think you 'll get a kick out of it. ’ Just shows that we 
never know what effect this course might have on others!! :-) ” 

Table 3 also shows through the whole course, there were 15 messages that facilitators provided 
‘criticism,’ all of which appeared in the first half of semester. Some conveyed facilitator’s confusion and 
frustration over technical problems, including those that might be caused by the change since summer, or 
facilitator’s uneasiness about participant’s low completion rate “It's been frustrating because of participants 
who didn't complete.” Moreover, a few facilitators expressed they could not reach administrative and 
technical support, such as the Help Desk. Following are a series of criticism at the end of October. 

“I just wanted to pass two things along.... 1. Severed of my participants have been complaining 
about the help desk ....that they have not been returning phone calls nor have they been able to 
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reach a person during the hours posted. 2. It seems that there has been many more technical 
problems than any other section ( this is my third section), is there a reason for that? Participants 
cannot get to the site, they cannot get a quiz or it does not post until several tries, they cannot get 
to links. ” 

“This is my fourth section of the course and my participants have had more technical issues this 
time than before. Initially, I thought it was growing pains 

with our district network, but we have supposedly solved those problems, yet participants are still 
complaining about various technical problems. Any insight?” 

“1 have to agree, this is the third time I have been a facilitator and there have been many 
technical difficulties... could this be because of the newly designed site? 

“ That is actually what I was thinking, because they resigned the site right after the end of the last 
section .... before that my participants did not have any of these problems ... that is a good point!” 
Facilitator’s Activities and Contributions 

One great use of the discussion forum was to ask for help, report to the FOR-PD program 
coordinator about problems and errors, and respond or give suggestions to questions raised by other peer 
facilitators (see Table 4). Facilitators’ posts of activities, similar to their comments, were posted more 
frequently in the first half of the semester than in the later half, with a ratio of 43:18. 



Table 4 Facilitator’s Activities in the Discussion Board in Phase 2 



Facilitator’s Activities 


Sept. 


Oct. 


Nov. 


Dec. 


ask for information; seeking help including how they could 
solve problems for participants’ 


X 


X 


X 


X 


report to FOR-PD UCF program coordinator and discuss 
with peer facilitators about technical, administrative and 
content problems and errors in different aspects, including 
access to quizzes and participants’ course pacing. 


X 


X 


X 


X 


respond/give suggestions to questions from peer facilitators 


X 


X 


X 


X 


Total Monthly No. 


27 


16 


11 


7 



Examples of comments from facilitators asking for information or seeking help, including how they could 
solve problems for participants, follows. 

“Could you please send me more brochures? 1 want to share them with the faculty again. Some 
people have shown an interest in taking the course. Is it possible for someone to sign up now for 
the fall course?” 

“I know this information is in my manned, but I'm at school. I need a refresher on how to close a 
session, ” 

“ How do you manually go back in and change the score? 1 need to fix a few of my participants’ 
scores. ” 

“ Since this is my first time facilitating the course, I'm not sure where I should post general 
feedback to the discussion. Do I create a new message in lesson 1 or 
somewhere else? Most of my participants have completed lesson 1. I would like to post a 
generalization about the discussion & test. If I have any specific comments to participants I will 
send it directly to the participants. Thanks for your help with this. ” 

“I have severed participants who are getting the following message when they go to take 
quizzes. ..I have emailed them about the pop- ups. ..is there any other reason it may be this way?? 
Here is the question l received: Susan, this past weekend I was on a different computer. 
However today is Monday and I am back on the computer that I have been using. I still 
cannot take a quiz. In fact, all of the quizzes are marked unavailable. 

I'd like to ease these participants' troubles. ” 

An example of a response to a question raised by a peer facilitator (for example, a response to the last 
question in the above paragraph) was phrased, “sounds crazy, but has she taken the survey? 1 had a few 
contact me about not being able to take lesson 2 quiz, but they hadn 't taken the survey. As soon as they did, 
all was okay. ” 

Following is a series of posts concerning accessing quizzes which provides an example of 
reporting correspondence with FOR-PD UCF program coordinator and discussion with peer facilitators 
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about problems and errors. 

“I have a participant that can not access quiz 3. I have called the help desk , she has called the 
help desk, I have tried her in every way that I can. She has submitted lesson 2 's quizzes and they 
have been graded. She still cannot access the quizzes. Now she is considering dropping from 
FOR-PD. What can / do ? We can 't figure out why she can 't access the quizzes. The Help Desk sees 
no reason why she couldn 't access the quizzes. I am at a loss!!!! ” 

“Has she done all the quizzes prior to Lesson 3 and the pre-course survey and have grades been 
posted for all? I do know that it will not let you skip a quiz. Has she tried to use another 
computer, some of my participants are having difficulties with their school computers” . 

“I had a participant who couldn't do the quizzes because the pop-ups were turned off on her 
computer. That's yet another thing to check.... ” 



Evaluation Summary 

The investigation of facilitator’s discussion board revealed that the overall pattern of facilitators’ 
use of the forum matched the expectations and requirements as indicated in the Request for Proposal (RFP) 
of Florida Department of Education and the UCF proposal for FOR-PD system. Facilitators’ activities and 
contributions ranged from giving technical support, monitoring of content including assignments, rubrics 
and course pacing, to facilitating interaction between FOR-PD Help Desk and participants. Moreover, 
facilitators generally provided encouragement for the revised FOR-PD course (phase II), including its new 
layout such as the facilitators’ forum and its new content such as the rubrics. However, some facilitators 
still had certain confusion and frustration over the change of the course, particularly about the access to 
quizzes and other technical problems, and about the availability of Help Desk in terms of administrative 
support during the first half of the semester (September and October 2003). 

Compared with the facilitator focus group report done May 2003, which provided a broad picture 
of FOR-PD system from facilitators’ perspective as the preliminary data for this evaluation, the discussion 
boards revealed to the current researchers how facilitators played their roles and helped participants 
enhance learning in a much greater depth. In addition, the ‘time -series’ analysis of the discussion board has 
shown that FOR-PD was getting better over time although confusion and problems still existed in phase II. 
To be specific, good themes had been maintained, as seen in both the focus group data in phase I and the 
data from the discussion board in phase II. For example, the general tone of responses from both data 
sources indicated that facilitators appreciated the current research-based content, and the database of 
materials and resources developed as part of FOR-PD. In addition, facilitators did not mention the WebCT 
grade book as their technology program any more in phase II. Neither did the facilitators suggest that 
interaction among participants were inhibited in this section as in the focus group report. To sum up with 
one post in the facilitator’s discussion board, “I am surprised there is so little activity on the discussion 
board this section. I guess so many of us are ‘old-timers’ that most of the questions have been answered.” 

Moreover, a trend was also found at different points of phase II, that is, more activities and 
comments of facilitators appeared on the discussion board in the first half of phase II than the later half in 
November and December, which implied that facilitators had helped participants solve their learning 
problems in a timely fashion. 



Recommendations 

The findings from the facilitator’s discussion board and the comparison made with the previous 
facilitator focus group also showed several ongoing problems in FOR-PD program. Following are a few 
recommendations to address them and a couple of suggestions for the final qualitative evaluation of FOR- 
PD program planned for May 2004. 

The Help Desk currently has a goal of addressing and resolving technical problems within a 24- 
hour period. Careful attention to continuing quick response should be monitored to ensure efficient and 
effective support in response to problems encountered by FOR-PD facilitators and participants. 2 

Facilitators should not only be updated and familiarized before changes are made in the FOR-PD 
system to ensure they are comfortable with and understand how the changes will impact the course and can 
thereby be more effective in assisting participants, but they should also be reminded about the changes — 
for example, the changes of access to quizzes caused some confusion with facilitators. 3 

While FOR-PD has been overall effective in using the online system, various technical problems 
have been frustrating to some participants and facilitators, which may or may not have been problems 
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within FOR-PD itself. It is suggested that attention to improving the technical aspects of FOR-PD and 
researching and implementing ways to make the technology more user-friendly should be continued. 

To understand the factors that have an impact on FOR-PD better, additional qualitative analyses 
should be conducted including telephone/on-line interviews and/or focus groups with five key audiences: 
school districts, participants, facilitators, Florida Department of Education staff, and FOR-PD content 
contributors or course designers/instructors. 
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Appendix: Data Analysis 

Robert Yin's book. Case study research: design and methods (2nd ed.) was used for the design for 
this qualitative evaluation and one of its dominant mode of case study analysis, the combination of ‘pattern - 
matching’ and ‘time -series analysis’ was applied to analyze and explain facilitators’ perceptions and 
experiences of FOR-PD phase II. Pattern matching “compare an empirically based pattern with a predicted 
one (or with several alternative predictions) for dependent variables’’ (Yin, 1994). In this case, the outcome 
of the use of facilitators’ discussion board was matched with the requirements of the RFP of Florida DOE 
and the focuses of the UCF proposal for FOR-PD program. On the other hand, time -series analysis or “the 
match between a trend of data points compared with a theoretically significant trend specified before the 
onset of the investigation’’ (Yin, 1994) was applied, too. In this paper, data points were set in each month: 
September, October, November and December; time -series analysis also included a comparison of the 
outcome of the investigation into the facilitator’s discussion board with the facilitator’s focus group report 
done May 2003. 

Moreover, with the qualitative software Nvivo Revision [1.3], automatic coding of the data was 
used in addition to hand coding. Firstly, all 161 messages were copied and imported to the Nvivo system, 
forming six major documents representing different genre of the data: Facfocus (the facilitator focus group 
report), Facilitator E-community, Problems, Suggestions and Comments, Success Stories, and 
Miscellaneous. Next, dozens of codes were developed following the basic guideline of the role, time, and 
content of the message, forming three dimensions in the coding process. By reviewing the data and the 
codes, or namely, ‘nodes’ as termed in the Nvivo system, the codes indicating the role of facilitators and 
those explaining the content of the message were combined into two big categories: facilitators’ 

‘comments’ and ‘activities’. Later, using cross-search of the Nvivo system, some within the two 
dimensions of ‘time’ and ‘comments’, and others within the two dimensions of ‘time’ and ‘activities’, 
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several ‘trees’ of the ‘nodes’ as termed in the Nvivo were formed. Finally those ‘trees’ of codes became the 
main themes of the data and were quoted as the findings of the paper under the two big categories: 
‘facilitator’s comments’ and ‘facilitator’s activities and contributions’. 
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